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ABSTRACT 



A screen peripheral system according to an embodiment of 
the invention includes a computing device for producing a 
main image and a touch-activated input device for generat- 
ing and displaying a composite image visible to a user. The 
composite image simultaneously includes a representation 
of at least one key, for example a QWERTY keyboard, for 
activating an input function, and the main image provided by 
the computing device. The keyboard representation prefer- 
ably is laid over the main image. According to one 
embodiment, the main image is an output image generated 
by an application being executed by the computing device. 
Other touch screen systems and methods are also disclosed. 

30 Claims, 9 Drawing Sheets 
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TOUCH SCREEN SYSTEMS AND METHODS image and a touch-activated input device for generating and 

displaying a composite image visible to a user. The com- 

CROSS-REFERENCE TO RELATED posite image simultaneously includes a representation of at 

APPLICATIONS least one key, for example a QWERTY keyboard, for 

... , .5 activating an input function, and the main image provided by 

The subject matter of this apphcatton is related to the ^ utin device ^ keyboird representation prefer- 

subject matter of commonly assigned U.S. Provisional b , fa wd m ^ ; 

Applications Ser. No. 60/024,780, filed Aug. 28, 1996, Ser. 2. . . . . • w. • , „ , , f .u 

No 60/028,028, filed Oct. 9, 1996, and Ser. No. 60/036,195, , ™* ""plements vanable-pDcel control to form the 

filed Jan. 21, 1997, all of which provisional applications are ,„ keyboard representation and to form the main image, caus- 

incorporated by reference hereto and priority to which is 10 P™ b * lected to fonD ,he ***>°f "^f ntot T 

claimed under 35 U.S.C. §119(e). * ected to fo ™ °* °>am image to be dependent on each 

v ' other. In other words, the keyboard-representation pixels are 

BACKGROUND OF THE INVENTION not independent of the main-image pixels. According to one 

embodiment, the main image is an output image generated 

1. Field of the Invention 15 D y an application being executed by the computing device. 
The invention relates to touch-sensitive input and output Various image-adjustment, mode-switching, zoom-up/ 

devices, and more particularly, to touch screens that simul- zoom-down and other embodiments associated with inven- 

taneously can display a keyboard-type image superimposed xion are also described, as are corresponding methods, 

on a background image. The invention also relates to other Embodiments of the invention have particular application 

features of touch-sensitive i/o devices. Embodiments of the 20 ^ wwraMe mmpoSin& deviceS) ^ as those available from 

invention have particular application to wearable-computing viA> Northfieldj Minn> Attention also is directed to the 

devices and environments, although non-wearable embodi- following us . patents> each of which ^ incorporated by 

ments are also contemplated. reference herein: U.S. Pat. Nos. 5,581,492; 5,572,401; 

2. Description of Related Art ^ 5,555,490; 5,491,651 and 5,285,398, all of which are owned 
It is known in the art to superimpose a keyboard over an by ViA, Inc. 

image that is output by an application being executed on a 

computer, i.e. to form a "phantom" keyboard, on a touch- BRIEF DESCRIPTION OF THE DRAWINGS 

sensitive input display device. U.S. Pat. No. 5,581,243 to Embodiments of the invention will be described with 

Ouellette et al, for example, which is incorporated by 3Q ct tQ tfae figureSj in which Jike reference nurne rals 

reference herein in its entirety, describes a system for denQte ^ elements> md m which: 

displaying a simulated keyboard on a touch-sensitive mrt „ . , , . 

uiapiaymx a xia j t . pj G \ & A perspecuve view of a wearable computing 

display, without occluding from view an underlying apph- . * * v . ^ 

\ 3 a * * •_. • environment according to an embodiment of the invention; 

cation output image. An output image is generated and « & 

displayed for a first period of time at a first luminous 3S FIG. 2 shows a keyboard representation according to an 

radiation intensity having a predetermined persistence. A embodiment of the invention; 

simulated keyboard image is generated and displayed for a FIG. 3 shows a main image according to an embodiment 

second, shorter period of time at a second, lower luminous of the invention; 

radiation intensity having the predetermined persistence. fiq 4 shows a composite image according to an embodi- 

The keyboard and output images are generated in ^ mc nt of the invention; 

alternation, and the amount of time each is displayed is piG 5 shoWfi a sec0Qd composite image accor ding to an 

controlled, to try to create the illusion that both images are embo diment of the invention; 

being displayed continuously. nQ 6 ^ a flow chart fihowin opcrational stcps accord ing 

The system described in Ouellette et al., however, is tQ ^ cmbodimcnt of thc mvcnt ion; 

believed to suffer a number of disadvantages. First, it is 45 , 

believed that flashing between application and keyboard 7 shows a reduced-key keyboard; 

screens, as the reference discloses, would significantly FIG. 8 shows a touch screen according to an embodiment 

reduce the refresh rate achievable relative to a continuously of me invention; and 

displayed application output (or a keyboard) image. This FIGS. 9-13 show touch screens having a handle, accord- 
reduction could cause considerable detriment to the viewer 50 ing to embodiments of the invention, 
and potentially could confuse the displayed interpretive nPSPRTPTrnN OF PHFFFRRFD 
features. Second, it is believed that displaying the keyboard DETAILED D ^^?™N i OF PREFERRED 
and the application output in alternation requires additional EMBODIMLN la 
hardware features for proper functioning, for example a Fp fapdimgnts of the invention oyA-WidejippUc^p ji ^ ji^ 
second frame buffer for switching back and forth between S5 number of comgu^g^ejovgonments. The invention is par- 
the screens. Additionally, although this feature is not dis- TicuTart3Ta^ 

closed or apparently contemplated in Ouellette, using the SScusse^a^oye, jn which portabMy jii^compactness are 

Ouellette device in a wire less- transmission (e.g. RF) mndamentalco jisid^ 

environment, for example in a wearable -computing context, to / !a^mImbe7pf differejjtjnput/output devices. For example, _^ 

would require e.g. twice as much information to wirelessly ^ ^bodimel^brt 

be transmitted and therefore would greatly increase the full,. partial^JrejJh^ed-key, .alphajiumeFic,Tjor^nj>nA ? 

required bandwidth. ^phanumeric.keyboards !.j»mprising. one or- more^'keys,", 

i^iUtfrns/^contact^zones'^or-lhe-like r-The invention-also^ 

SUMMARY OF THE INVENTION ^^^^^of^^^^d,^,^ 

To overcome the above and other disadvantages, a screen 65 ganuaU YQr.audibly^^ 

peripheral system according to an embodiment of the inven- j]£t-OTeciisjre<niirmg^ 

tion includes a computing device for producing a main user^srfiriger. Proximity 
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inputs are among the functionalities encompassed by 

^ P^ll^^^rc , i^ aT^yj-^l^rd^Snatgs , p articular^ 
„ ^poinT^r^z^e^prrthe^ display, is contemplated foruse^with^ 

^T^^e^bodiments will ,be described^dtb, respect to. touchy 
t screens, keyboards and tpuctynput^the .invention isjnje^dec^ 
lojje^roadly^jntcrpreted as needed Ind ho f necessarily 
limited to those particular embodiments/ v 

A d d i t io p a 11 v y.arioju s ^ ima ge,s^Qan^be^cona^ine . d/ JQ 

supermiPQsedjicrorfin g^^^ In 
primary embodiments, the superimposed image is that of at 
least one key, e.g. a keyboard. The background image is that 
output by an application executed by a computing device, 
for example a graphics, spreadsheet, word-processing, etc. 15 
application for use in private, commercial, military, public- 
service fields or other fields. However, either the background 
image or the superimposed image can be of other types; for 
example the background image, or both images, can be 
provided by a processor or storage device associated with 
the system, instead of by an application program per sc. 
Either or both image can be the subject of an enlargement or 
reduction process, to provide composite images having 
elements of different sizes. MorcJthanJyyjjJmagcs^can^ 
combined according to the invention as well; any combina- 

-embodiments wili be described herein witff ; reference ? to 

^fipJi^ii^^^PJi^^SSS, keybbards^c^ 

r intended to be broa^ly^njerpreted as needed ana^qUiecr^Q 
es saaiv ' llmited Jo t ho se Vi ^icula rlemb 

Touch screen keyboard overlays according to embodi- 
ments of the invention can be selectively turned on and off 
by a user of an associated personal computing device, for 
example a wearable personal computer. A representation of 35 
a keyboard, as will be described, lies over a main image on 
the touch screen, the main image preferably being formed in 
connection with a main software application. A keyboard 
overlay according to the invention eliminates a physical 
keyboard or the necessity of having a split screen with a 40 
touch keyboard on a lower portion and a main screen 
squeezed into the remaining upper portion, for example. A 
keyboard overlay according to the invention also reduces 
and/or eliminates the space-conflict problems that can arise 
when a superimposed keyboard occupies the same portion of 45 
the display screen as a critical portion of the application 
output image. 

Keyboard overlays according to the invention are espe- 
cially advantageous for very small screens, because of the 
relatively large and accessible overlay "buttons" (keys) that' 50 
are provided, along with the ability to see all of the data 
behind the keyboard at the same time. 

Embodiments of the invention are especially advanta- 
geous in situations where voice -based input is undesirable. 
Although voice-recognition technology is making signifi- 55 
cant advances, it still requires verbal input, which can be 
disruptive to a meeting, for example. Verbal input also 
potentially announces to all within earshot of the speaker the 
exact notes, etc. that are being entered. According to 
embodiments of the invention, therefore, the user inputs 60 
directly to the machine in complete silence. Still, voice - 
based input may be desirable in many cases, so optional 
switching between voice input and keyboard-overlay input 
is also contemplated. This can be activated by a physical 
button or key on the side or other surface of the touchscreen 65 
or associated device, or by a virtual button or key on the 
touch-sensitive portion of the screen itself, or by voice. 
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Keyboard overlays according to embodiments of the 
invention can take a number of forms. For example, a 
solid-line or dashed/dotted-line overlay is contemplated, 
providing outlines of each key. Alternatively, and selectively 
as chosen by a user, each key can be represented by a dot, 
or as a letter-area zone between intervening markers such as 
x's, dots, etc. Ideally, the representation of the keyboard is 
provided faintly on the standard screen, so that it and 
data/images from the main program can be seen simulta- 
neously. Variable-pixel controls, described in more detail 
below, can be provided to change the thickness, brightness, 
and/or dotted-ness of the keyboard representation. In many 
cases the keyboard overlay can be extremely faint. 
Additionally, a color difference can be imparted to the pixels 
of either the keyboard overlay or the underlying image, or 
both, to highlight the difference between the two. 

The characters of the keyboard themselves, e.g. the letters 
associated with each key or zone, can be turned off if the 
user has memorized the keyboard layout, for example. The 
keyboard can be active without being visible at all, for the 
user who can type effectively without even seeing represen- 
tations of the keys/zones themselves. 

Conin e* adjug^ pnt huttnns are contemplated, preferably 
rep resented on ^ the touch screen itself but^possib ly on a 
housing of the touch screen, to adjust contrast between the 
keyboard *nri the main screen. A keyboard on-off button is 
also contemplated. Alternatively, these features can be con- 
t rnjls H hy voire 

Variable-pixel controls 

In accordance with embodiments of the invention, 
software-based variable -pixel controls are provided to deter- 
mine and control which pixels of the touch screen will be 
used for displaying the keyboard representation and which 
pixels for displaying the main image. In some cases, each 
pixel of the screen is 100% dedicated to either the keyboard 
or the main image. In other cases, touch screen pixels may 
be dedicated to both the keyboard and the main image, 
producing a "blended" effect as will be described. 

Merging the image of the virtual keyboard with the 
normal output image on the display according to the inven- 
tion can occur by a variety of methods and programming 
schemes. Using most current video graphics controllers, 
however, it is believed a highly efficient method available to 
perform the merging of images is to use bit- block or 
bit-block-type transfer operations, i.e. BitBlt operations. 

BitBlt operations provide an efficient method of perform- 
ing logical combination of up to three sets of pixels on 
raster-based display devices. According to embodiments of 
the invention, BitBlt operations are used with the following 
three sets of pixels: 

(1) The original pixels on the display, i.e. the image that 
would be presented in the absence of the virtual key- 
board. 

(2) The pixels representing the image of the virtual 
keyboard. 

(3) An image mask, allowing control of which pixels 
within the virtual keyboard will be merged with the 
original display pixels. 

Using BitBlt operations, the virtual keyboard can be 
combined with the display using a variety of effects. The 
following table summarizes the typical operations that can 
be performed: 



Page 12 (ANelsonl, 02/25/2001, EAST Version: 1.01.0015) 



5 



6,121,960 



6 



Source (S) 


11001100 






Destination (D) 


1 0 1 0 1 0 1 0 


Boolean 




Mask (M) 


11110000 


Operation 


Operation 


Result 


nnnnnnfirt 
UUUUUUUU 


Q 


Blackness 




000 100 0 1 


-<S|D) 


Sot source erase 




0 0 110011 


-S 


Not source copy 




0 1000 100 


S&-D 


Source erase 




0 10 10 10 1 


-D 


Destination invert 




0 10 110 10 


M " D 


Mask invert 




0 1100110 


S * D 


Source invert 




10 001000 


S & D 


Source and 




10 1110 11 


-S|D 


Merge paint 




1 1 0000 00 


M & S 


Merge copy 




110 0 110 0 


S 


Source copy 




1110 1110 


S|D 


Source paint 




11110000 


M 


Mask copy 




111110 11 


M|~Sp 


Mask paint 




11111111 


1 


Whiteness 



Additionally, those of skill in the art upon reading this 
disclosure will comprehend that similar logic functions can 
be applied to the color-differentiation capability described 
earlier; such functions are not limited to black/white/grey 
capabilities. 

The logical operations described herein each preferably 
have a different blending or merging effect. Although some 
of them do not have substantive effect, e.g. they just copy the 
source in to the destination or just fill it with zeros or ones 
and are ignorant of what the source and destination have, the 
large majority of these operations can be used to create a 
number of different effects to determine how to mesh the 
keyboard image and the application output image together. 
Additionally, the various logical operations can be combined 
as needed to produce an even greater variety of visual 
effects, e.g. harshness/intensity of one image with respect to 
the other. 

Two examples of the myriad of meshing possibilities 
contemplated according to the invention will now be 
described. In the first example, the keyboard is blended with 
the original display application-output image by using 25% 
of the pixels to represent the keyboard and 75% of the pixels 
to represent the original display image. In the second 
example, the two images are more truly blended together, 
with each pixel of the final composite display image receiv- 
ing a contribution from both images. For the second 
example, the composite realized image can appear as e.g. a 
blended shadow from the original images. For both 
examples, it is assumed according to one embodiment that 
the graphics controller of the computer system is using an 
8-bit bit plane, which is typical of most personal computers 
supporting 256 colors per pixel. Of course, those of ordinary 
skill will readily be able to adapt these examples for use in 
other current and future graphics controllers, bit planes, etc. 

In the first example, where the keyboard is blended with 
the display by using 25% of the pixels to represent the 
keyboard and 75% of the pixels to represent the original 
display image, each display update preferably is performed 
in three steps. The first step removes that portion of the 
keyboard image that is not to be seen, i.e. it eliminates 75% 
of the keyboard image. The second step removes that portion 
of the main display image that is not to be seen, i.e. it 
eliminates 25% of the main display image. The third step 
merges these two images. The following pseudocode repre- 
sents one embodiment of the associated processing used 
according to this example: 



15 



reduced_kbd BitBlt 


(source 


-> keyboard image, 




destination 


««> keyboard image, 




mask 


-> [OxFF; 0x00, 0x00, 0x00, 






OxFF, 0x00, 0x00, 0x00, 




operation 


...1 
-> merge copy); 


reduced_di£play BitBlt 


(source 


-> original display image, 




destination 


-> original display image, 




mask 


-> [0x00, OxFF, OxFF, OxFF, 






0x00, OxFF, OxFF, OxFF, 




operation 


...1 
»> merge copy); 


display :- BitBlt 


(source 


-> reducedL_kbd, 


destination 


«> rcduced_display, 




mask 


-> null, 




operation 


-> source paint); 



With the first example, for performance reasons the first 
BitBlt can be performed once at initialization and then 

20 stored for reuse in future display updates. Essentially, the 
process eliminates from the two images those pixels that are 
not desired for the single, final composite image that is 
displayed to the user. In this way, neither image contains 
overlapping pixels. In the combined image, therefore, each 

22 pixel is contributed 100% by either the keyboard or the 
display, but 0% from the alternate. 

With the second example, the two images are logically 
combined using a single BitBlt, according to one embodi- 
ment. The following pseudocode represents one embodi- 

30 ment of the associated processing used according to this 
example: 



display :° BitBlt (source «> keyboard image, 

35 destination -> original display image, 

mask »> null, 

operation -> source paint); 



Current embodiments of the invention employ the fol- 

40 lowing assumptions for performance reasons. It is contem- 
plated that embodiments of the invention are for typically 
available operating systems, ones that support multi-tasking, 
with each application having its own thread of control, and 
a graphical user interface using windows as the visual 

45 representation of each task. Examples of such operating 
systems are Microsoft Windows 95, Microsoft Windows NT, 
SCO Unix, and Sun's Solaris operating system. Embodi- 
ments of the invention are also for typical video graphics 
controllers, ones that support bit -block transfer (BitBlt) 

50 operations and DMA between processor memory and video 
memory. Graphics controllers from Chips and Technologies, 
S3, NeoMagic, Trident, and other companies all support 
these operations, for example. 
Embodiments of the invention should be useable without 

55 necessarily relying on these assumptions, as hardware and 
software technologies evolve, for example (such as with 
multiprocessors used in one device, for example). Even 
currently, implementation in environments other than those 
described above is contemplated according to the invention, 

60 even though implementation in at least some of those 
environments may be more complex and subject to slower 
operation. 

Thus, according to embodiments of the invention, and as 
shown in e.g. FIGS. 1-5, screen peripheral system 5 accord- 
65 ing to an embodiment of the invention includes computing 
device 10 for e.g. executing a software application, and 
touch -activated input device 15 for generating and display- 
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ing a composite image visible to a user 20 of screen eliminated key representation with the pixel-eliminated 

peripheral system 5. The composite image includes, accord- main image to form the composite image. Alternatively 

ing to one embodiment, a representation of at least one key and/or additionally, as in the second example described 

for activating an input function, e.g. a keyboard-input func- above, the composite image can include a blended shadow 

tion. The composite image also includes a main display 5 of the representation of at least one key and the main image, 

image, for example the output image produced by the Computing device 5 can control the lightness/darkness of 

software application being executed by computing device the blended shadow by controlling the blending of pixels of 

10. The representation of at least one key is laid over the the at least one key representation and the main image, 

main image produced by the software application. - according to one embodiment. By changing the darkness of 

FIG. 1 shows screen peripheral system 5 in a wearable- 10 the pixels of the composite image, for example the keyboard 

computing environment, which as described earlier is espe- portion of the composite image, the keyboard can be made 

cially well-suited to embodiments of the invention. Embodi- to appear to "bleed through" the other image. Thus, the 

ments of the invention can also be used in connection with keyboard "steals away" appropriate pixels from the main 

a miniature, low-power, spread-spectrum "bodyLAN"-type image, and/or shades them differently in a selected manner, 

system allowing selective control based on proximity to a 15 to produce the keyboard image. 

user. With such a system, touch -activated input device 15 As will be appreciated by those of ordinary skill in the art 

can be in either wired or wireless communication with upon reading this disclosure, methods corresponding to the 

computing device 10 and/or other devices/systems, for above-described apparatus and systems are also contem- 

example LANs, WANs, etc. Wired and wireless communi- plated according to the invention. 

cation embodiments are also contemplated for use with 20 FIGS. 2-5 illustrate sample images and composite screens 

no n- wearable computing devices as well. Fiber optic, according to embodiments of the invention. FIG. 2 illus- 

electrical, infrared, RF and other data-transmission schemes tratcs a representation of at least one key — in this example 

between e.g. the disclosed input device and the wearable or a full QWERTY keyboard with several function/option 

other computer, or between any other elements of the keys/buttons. FIG. 3 illustrates a main image generated by a 

system, are contemplated. For example, RF transmission to 25 computing device. FIG. 3 should also be interpreted to cover 

keyboard or a wearable, desktop, laptop or other computer. the situation where the main image is an output image or 

Embodiments of the invention are especially useful with images of an application or applications being executed by 

tablet-type touch screens and other small, readily portable/ a processor or other element of the computing device. FIG. 

pocketable input/output devices. 4 shows a composite image formed in accordance with 

Embodiments of the invention also can be used in con- 30 Example 1 above, and FIG. 5 shows a composite image 

nection with heads- up and/or body -worn display devices, formed in accordance with Example 2 above, 
e.g. dual-screen stereo displays. A keyboard overlay accord- FIG. 6 illustrates a flow chart according to an embodiment 

ing to the invention, preferably a voice -activated and/or of the invention, identifying the generalized processing that 

voice-controlled overlay, appears on the display, allowing occurs to allow a virtual keyboard to be overlayed upon a 

the user to input data without looking down. Automatic 35 normal display. This processing maintains consistency of the 

typing features with voice -recognition can be provided, virtual keyboard image with the rest of the information 

whereby the user sees individual keys of the keyboard being being displayed. 

activated as the user speaks. Voice can thus be used as an After start step 100, a virtual keyboard image is created, 

intuitive keyboard approach for the spoken words/letters in at step 105. The keyboard is drawn as an overlay on the 

a hands-free environment. 40 current display in step 110, and then an event is awaited at 

Screen peripheral system 5 implements variable-pixel step 115. When a touch event occurs at step 120, the virtual 

control to form the representation of at least one key and to key touched is determined at step 125 and the keyboard 

form the output image, for example in the manner described event is sent to the current task/window at step 130. The next 

above. The variable-pixel control causes the pixels used to event is then awaited at step 115. 

form the representation of at least one key and the pixels 45 If there is no touch event at step 120, it is determined 

used to form the output image to be directly dependent on whether a pertinent window has been redrawn, at step 135. 

each other, in a manner that for example is also described If so, the area of screen redrawn is determined at step 140, 

above. In other words, unlike the Ouellette reference dis- and then it is determined at step 145 whether the area of 

closed above, the pixels chosen to represent the images are screen redrawn overlaps the keyboard image. If not, the next 

not independent. 50 event is simply awaited at step 115. If so, the keyboard is 

The variafrl A -fHf A i1 ranfr^lran . pro^n^e.^-on'ra^-adjH^. (re)drawn as an overlay on the current display at step 110. 

j3^U3etween*me.represeaUt&^ It is also determined whether a window is deleted at step 

.joutput | image_ produced„b ^ 150, and whether a window is created, at step 155. If the 

contrast ^justment canjoclude^ch answer is yes, steps 140, 145 and 110 or 115 occur as 

^ruckness, P bn^^ described above. If the answer is no, the next event is (still) 

iatipnj)f^tjejau^nejtey.^ awaited at step 115. 

As referenced above, according to one example comput- Mode-switching 
ing device 5 eliminates X% of the pixels of a full represen- Recordin g to on e embodiment, jfa re^resentation.otakey 
tation of at least one key, X% representing that portion of the _or button Js,held "down" for a set lime,.foL example.one 

keyboard that is not to be seen in the composite image. In the 60 seco nd, a .^cursor appears at the -point of the depressed-key 

first example described above, X=75, although X can have and the remainder of the keyboard (optionally) ^disappears. a 

a variety of different values to suit a particular display or B^^raggmg a finger along the touch screen, the user can 

purpose. Computing device 5 also eliminates (100-X)%, e.g. move the cursor to a desired location. According to one 

25% as described above, of the pixels of a full application embodiment, upon entering the cursor mode a number of 

output image. Here, (100-X)% represents that portion of the 65 new keys/buttons (or representations thereof, as above) 

application output image that is not to be seen in the appear on a side of the screen, for example, cut, paste, copy, 

composite image. Computing device 5 merges the pixel- delete and/or hold buttons. An appropriately placed mouse 
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button is also contemplated on the screen, e.g. in a comer of keyboards, by superimposing the keys of the sub-keyboards 
the screen, to be accessed by a user's thumb. A convenient on top of each other. According to one example, the alpha- 
holding mechanism for the screen, allowing easy access by betic sub-keyboard is superimposed on the numeric and 
the user's thumb, is described below. function sub-keyboards. For numeric input, the numbers are 

In use, a user types on the keyboard overlay for a desired 5 printed not at the top of each key but where multiple 

time, and then holds one finger down for a set period of time adjacent letter keys meet, at the interstices between the keys, 

to exit the keyboard mode and enter the cursor mode. p our « quartcr » keys thus operate together to register the 

Selected text can be highlighted using the above-described desired mmhcrf bllt cacn num b cr key is fuuNsizcd. 

mouse button in e.g. a comer or on a side of the touch screen. Bcncath ^ four lcUcr k associated ^ cac h number are 

and then the cursor can be dragged to hover over a cut button ^ . k Qr c[cmc ^ QT ^ ^ acd . 

or other desired button. Movement of the user s fineer f rp ■ . t_ * * t_ \ L * 

becomes equivalent to movement of the mouse when The * aUon of a .f*™; k Dumbcr ° f ™« areas by the user's 

finger is held down, and the touch button for the mouse is in fin f r rcptenng the desired number This technology 

the corner of the screen so that it is accessible by the thumb. rcduccs * c . ovc « u SI2 f of f thc keyboard for example to 

Upon exiting the cursor mode, the user's fingers are again credit-card ^Examples ^of these types of key layouts are 

used for typing. 15 snown m u - s - Pat - No - 5,612,690, among others. U.S. Pat. 

Reduced-key embodiments No - 5,612,690 is incorporated herein by reference. 

According to one keyboard overlay embodiment, the Superimposing the keys of various sub -keyboards on top 

number of keys of a virtual QWERTY keyboard is reduced of each other, such as an alphabetic sub-keyboard superim- 

to e.g. twelve, nine of which are letter keys and the other posed on a numeric and/or function sub-keyboard, can be 

three of which are shift, delete and select keys, for example. 20 combined with the reduced-key concepts disclosed above 

Note FIG. 7, showing one type of reduced-key keyboard with respect to FIGS. 7-8, to further reduce the size of a 

200. Each letter key represents multiple letters, for example keyboard or keyboard overlay according to the invention, 

three letters or letter combinations. A word-recognition With the layout of the FIG. 7 embodiment, for example, 

program discerns the word intended to be typed by hitting number and/or function keys would be placed at the physical 

the letter keys, e.g. upon depressing the select (or space) 25 or virtual "interstices" between the three-letter keys shown, 

keys. Reduced-key keyboard patterns enhance miniaruriza- The resulting composite image would then be superimposed 

tion achievable according to the invention and provide other over an application output in a manner described previously, 

advantages. for example. 

According to one embodiment of the invention, a Zoom embodiments 

reduced-key keyboard is incorporated as a virtual or physi- 30 According to other embodiments, zoom-up/zoom -down 

cal touch layout 210 on a standard touch screen/slidepad capabilities are incorporated through a touch screen or other 

mouse 220, as shown in FIG. 8. A program monitors the touch-sensitive input device, for example in a manner com- 

touch screen 220 continually, detecting when a virtual option patible with the above-described technologies. Once an 

button 230 or physical option button 240 is selected to appropriate zoom mode is activated, e.g. by virtual or 

change the mode of operation from a standard touchpad 35 physical keys, contact of one, two, three or more of a user's 

mouse ("relative" mode) to a keyboard-input touchpad fingers with the touch -sensitive surface causes activation of 

("absolute" mode). Upon entering the keyboard mode, a key an appropriate number of sensing areas and/or "zones" 

pattern overlay (e.g. a letter keyboard, number pad, and/or thereon to control screen size and/or the amount of magni- 

individual keys) according to the invention appears on the fication displayed. According to one example embodiment, 

touchscreen directly and/or on an associated screen. It is 40 one -finger contact causes normal screen size and/or 

then detected which key of the keyboard the finger of the magnification, two-finger contact causes a zoom-up to 

user is lapping on to generate letter/word or other input. double size/magnification, three-finger contact causes triple 

Using a reduced-key keyboard in connection with a standard size/magnification, and so on. Zoom-down capabilities func- 

touchpad mouse, with mode -switching as described above, tion can be configured similarly. Sequential step -up or 

provides significant display-miniaturization and key- 45 step-down is not required; for example triple-size/ 

enlargement advantages, as well as other advantages such as magnification can be reduced to normal size/magnification 

an increase in one- finger typing speed. merely with the touch of one finger, without entering a 

According to another embodiment, the virtual or physical double-size/magnification mode, 

select or other buttons 230, 240, or another area on the touch Thus, the user is spared the burden of going to a file menu, 

screen itself, can be depressed multiple times, or held down 50 pulling down to the zoom feature and then selecting zoom-in 

for a longer period of time, to enter and/or exit various and zoom-out functions using e.g. the mouse. A virtual, 

submodes of operation. The first touch can cause a change physical, or voice- activated zoom function select button is 

from a relative mode to an absolute mode, and subsequent activated, and, simultaneously or thereafter, the touchscreen/ 

touches to absolute submodes such as alphabet, numeric, slidepad itself determines how much of the touchscreen is 

punctuation and/or other sub-modes. For example, if a user 55 covered at any one time by the user's fingers (or other 

desires to input e.g. punctuation while in a standard relative pointing device), providing a specific screen size/ 

mouse mode, the user taps the appropriate button or buttons magnification related to the number of fingers being used, 

twice, first to enter the absolute mode and then to enter the According to a more specific embodiment, appropriate zoom 

punctuation input sub-mode, enters the appropriate functionality occurs based on the number of sensing areas or 

punctuation, and then taps the button or buttons twice again 60 contact zones activated on the touch-sensitive input device, 

to return to an absolute keypad mode. One or a plurality of A particular area of a screen chosen can "zoom up" at the 

buttons can be used to switch between relative and absolute user as that area is touched. 

modes, and, within e.g. the absolute mode, to choose a These features have special relevance to portable, wear- 
keyboard, number pad, punctuation input sub-mode or other able computing, in which miniaturized screens are used, for 
option. 65 example on the wrist, and generally are not full -sized. To use 
It is also known in the art to reduce the size of a full existing software, a user may want to see screen features in 
keyboard, with alphabetic, numeric and function sub- larger, zoomed-in format at certain times and, at other times, 
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in a zoomed-out format in refer ence to how items look on 
an overall page. Flip and rotate commands, either voice- 
based or touch-based, can also be implemented to improve/ 
augment the user's view of the displayed information. 
Other hardware embodiments 5 

One problem with prior touch screens is that for miniature 
computers and miniature screens, the user's hand is often 
bigger than the screen itself. According to the embodiments 
of the invention, therefore, retractable handle 260 (FIGS. 
9-10) slides relative to touch screen 270 between extended 
and retracted positions, as shown at 280, to provide a 
convenient holding mechanism at either the left or right side 
of touch screen 270. To shift handle 260 from the left side 
to the right side, or vice versa, pivot 285 is provided to swing 
the handle over the touch screen, as shown at 290. In the 
retracted position, the handle catches on the back side of the 15 
touch screen, permitting the pivoting motion. Upon travel- 
ing to or remaining in an extended position, a ball or similar 
projection snaps into a lateral groove or equivalent mecha- 
nism to prevent pivoting. Thus, embodiments of the inven- 
tion are significantly advantageous for use by both left- 20 
handed and right-handed individuals, as shown in FIG. 10, 
as well as those who need to use alternative hands for 
different tasks. Handle 260 can also be used as a carrying 
handle (FIG. 11), as part of a necklace embodiment (FIG. 
12), or in a wristband embodiment (FIG. 13) using e.g. a 25 
VELCRO strap. Additionally, according to another 
embodiment, a double^sided touch screen is provided, with 
the capability of flipping the screen by voice or by touch to 
accommodate left-handed or right-handed gripping of the 
touch screen by the handle without pivoting the handle from 30 
one side to the other. Multiple screen bleedthroughs, e.g. 
from one screen to another, on multiple screens, are also 
contemplated. A person with a folding screen or two displays 
can have them on more than one screen. 

As indicated earlier, embodiments of the invention have 35 
application in voice-recognition and/or hands-free embodi- 
ments. A microphone/speaker can be built in the face of the 
touch-sensitive input device, permitting a user to look at and 
talk to the device without having a separate microphone/ 
speaker. Voice commands can be used to change turn key- 40 
board overlays and/or voice-input modes on and off, and to 
change contrast, keyboard representation types, keyboard/ 
cursor modes, etc. Voice commands also can be used to 
activate the touch screen itself, so that it doesn't turn on by 
e.g. being depressed in a user's pocket, for example. 45 
Other features 

Sound and flashing can also be used in connection with an 
area to highlight where and when a user has touched the 
input device. Merging program use can be incorporated to 
highlight certain program outputs that are active and/or 50 
de-emphasize those that are inactive, for example by using 
a 25% pixel ratio for de-emphasis. As a user switches from 
one active program to another, their bleed-through intensi- 
ties can shift and change accordingly. Multiprocessing be 
incorporated, wherein separate screens are run by separate 55 
processors, one for each screen. Keyboards can be moved, 
and percentage areas changed, by dragging. Although pre- 
ferred embodiments have been described with respect to two 
images, any number of images can be combined/blended 
according to embodiments of the invention as long as their 60 
visual distinctions are preserved to the user's eye. 

It is contemplated that features disclosed in this 
application, as well as those described in the above appli- 
cations incorporated by reference, can be mixed and 
matched to suit particular circumstances. Various other 65 
modifications and changes will be apparent to those of 
ordinary skill. 



What is claimed is: 

1. A screen peripheral system, comprising: 

a computing device for providing a main image; and 

a touch-activated input device for generating and display- 
ing a composite image visible to a user of the screen 
peripheral system, the touch-activated input device 
comprising a plurality of pixels, the composite image 
simultaneously including: 

a representation of at least one key, the representation of 
at least one key activating an input function; and 

the main image provided by the computing device, the 
representation of at least one key being laid over the 
main image; 

wherein the screen peripheral system implements 
variable-pixel control to form the representation of at 
least one key and to form the main image, the variable - 
pixel control causing pixels selected to form the rep- 
resentation of at least one key in the composite image 
to depend on and be activated simultaneously with 
pixels selected to form the main image, such that the 
main image and the representation of at least one key 
are displayed simultaneously to form the composite 
image; 

further wherein the variable-pixel control includes logical 
operators to provide different blending/merging effects 
such that individual pixels of the touch- activated input 
device can be dedicated simultaneously to both the 
main image and the representation of at least one key. 

2. The screen peripheral system of claim 1, wherein the 
variable-pixel control provides contrast adjustment between 
the representation of at least one key and the main image 
produced by the computing device. 

3. The screen peripheral system of claim 2, wherein the 
contrast adjustment includes changing at least one of the 
thickness, brightness, and dotted-ness of the representation 
of at least one key. 

4. The screen peripheral system of claim 1, wherein the 
representation of at least one key is a dotted-hne represen- 
tation. 

5. The screen peripheral system of claim 1, wherein the 
representation of at least one key includes a representation 
of a full keyboard. 

6. The screen peripheral system of claim 1, wherein the 
touch-activated input device operates in two modes, the first 
mode being a keyboard mode in which a user activates input 
functions by intermittently touching the input device, and 
the second mode being a cursor mode in which the user 
moves a cursor associated with the touch-activated input 
device, the second mode being entered from the first mode 
and/or the first mode being entered from the second by 
continuously touching the input device for a set time. 

7. The screen peripheral system of claim 1, wherein the 
computing device forms the composite image by merging 
the representation of at least one key with the main image. 

8. The screen peripheral system of claim 7, wherein the 
computing device merges the representation of at least one 
key with the main image by using a bit-block-type transfer 
operation. 

9. The screen peripheral system of claim 8, wherein the 
bit-block-type transfer operation performs logical combina- 
tions of three sets of pixels: 

(a) the pixels of the main image; 

(b) the pixels of the representation of at least one key; and 

(c) optionally, an image mask for controlling which pixels 
of the main image will be merged with which pixels of 
the representation of at least one key. 
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10. The screen peripheral system of claim 1, wherein each 
pixel of the touch-activated input device is contributed 
100% by either the pixels of the main image or the pixels of 
the representation of the at least one key to form the 
composite image. 5 

11. The screen peripheral system of claim 1: 

wherein the computing device eliminates X% of the 
pixels of a full representation of at least one key, X% 
representing that portion of the keyboard that is not to 
be seen in the composite image; 1Q 

further wherein the computing device eliminates (100- 
X)% of the pixels of a full main image, (100-X)% 
representing that portion of the main image that is not 
to be seen in the composite image; 

further wherein the computing device merges the pixel- 15 
eliminated key representation with the pixel-eliminated 
main image to form the composite image. 

12. The screen peripheral system of claim 1, wherein the 
composite image includes a blended shadow of the repre- 
sentation of at least one key and the main image, or a clear 
space around a blended area to highlight the area of blend- 20 
ing. 

13. The screen peripheral system of claim 12, wherein the 
computing device controls the lightness/darkness of the 
blended shadow by controlling the blending of pixels of the 

at least one key representation and the main image. 25 

14. The screen peripheral system of claim 1, wherein the 
computing device executes a software application, the main 
image being an output image produced by the software 
application executed by the computing device. 

15. The screen peripheral system of claim 1, wherein the 30 
touch- activated input device provides zoom-up/zoom-down 
capability dependent on the number of the user's fingers in 
contact with the touch -activated input device. 

16. The screen peripheral system of claim 1, wherein the 
touch- activated input device allows switching between rela- 35 
live and absolute input modes depending on the number of 
times or amount of time a user contacts a selected area of the 
touch- activated input device. 

17. The screen peripheral system of claim 1, wherein the 
touch- activated input device allows switching between ^ 
keyboards/alphanumeric sets depending on the number of 
times a user contacts a selected area of the touch-activated 
input device. 

18. A wearable computing system comprising the screen 
peripheral system of claim 1. 45 

19. A method of superimposing a representation of at least 
one key over a main image provided by a computing device, 
the method comprising: 

(a) using variable-pixel control to form a representation of 

at least one key, the representation of at least one key 50 
activating an input function, and to form the main 
image, the variable-pixel control causing pixels 
selected to form the representation of at least one key 
to be activated simultaneously with pixels selected to 
form the main image; and 55 

(b) generating and displaying a composite image visible 
to a user of the screen peripheral system, the composite 
image simultaneously including the representation of at 
least one key and the main image produced by the 
computing device, the representation of at least one key 60 
being superimposed on the main image; 

wherein the variable-pixel control allows individual pix- 
els to be dedicated simultaneously to both the main 
image and the representation of at least one key. 

20. The method of claim 19, wherein the step of (b) 65 
includes the step of merging the representation of at least 
one key with the main image. 



21. The method of claim 20, wherein said merging 
includes using a bit-block -type transfer operation. 

22. The method of claim 21, wherein (b) further includes 
combining three sets of pixels: 

(aa) the pixels of the main image; 

(bb) the pixels of the representation of at least one key; 
and 

(cc) optionally, an image mask for controlling which 
pixels of the main image will be merged with which 
pixels of the representation of at least one key. 

23. The method of claim 21, wherein (b) further includes: 

(c) eliminating X% of the pixels of a full representation of 
at least one key, X% representing that portion of the 
keyboard that is not to be seen in the composite image; 

(d) eliminating (100-X)% of the pixels of a full main 
image, (100-X)% representing that portion of the main 
image that is not to be seen in the composite image; and 

(e) merging the pixel-eliminated key representation with 
the pixel-eliminated main image to form the composite 
image. 

24. The method of claim 19, wherein the composite image 
includes a blended shadow of the representation of at least 
one key and the main image. 

25. The method of claim 24, wherein (b) includes con- 
trolling the lightness/darkness of the blended shadow by 
controlling the blending of pixels of the at least one key 
representation and the main image. 

26. A screen peripheral system, comprising: 

means for computing, the means for computing providing 
a main image; 

means for displaying a composite image visible to a user 
of the screen peripheral system, the means for display- 
ing also being for input to the means for computing, 
wherein the means for displaying comprises a plurality 
of pixels, the composite image simultaneously includ- 
ing: 

a representation of at least one input zone, the represen- 
tation of at least one input zone activating an input 
function; and 

the main image provided by the means for computing, the 
representation of at least one input zone being laid over 
the main image; 

wherein pixels selected to form the representation of at 
least one input zone are activated simultaneously with 
pixels selected to form the main image, such that the 
main image and the representation of at least one input 
zone are displayed simultaneously to form the com- 
posite image; 

further wherein individual pixels of the means for dis- 
playing can be dedicated simultaneously to both the 
main image and the representation of at least one input 
zone. 

27. The system of claim 26, wherein the representation of 
at least one input zone comprises at least one key of a 
keyboard. 

28. The system of claim 26, wherein the representation of 
at least one input zone comprises a contact zone. 

29. The system of claim 26, wherein the representation of 
at least one input zone comprises a button. 

30. The system of claim 26, wherein the means for 
displaying a composite image comprises a touch screen. 
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